Into the Crossfire

Evaluating the Use of a Language Model to Crowdsource Gun Violence Reports

Adriano Belisario, Scott A. Hale, Luc Rocher

The challenge

Using AI with crowdsourced social media data is a promising resource for human rights monitoring (Alhelbawy et al, 2020), but…

  • Keyword-based search leads to a high proportion of unrelated text.

  • Organizations have limited resources to deal with large volumes of data.

Background

Previous studies have explored machine learning models for human rights monitoring (Alhelbawy et al., 2020; Pilankar et al., 2022; Ta et al., 2022).

However, there was a gap in the literature regarding applications on…

  • Applications for Portuguese-language texts.

  • Evaluations of adopting these models in real-world settings.

Our work

We deployed an open-source language model to assist crowdsourcing gun violence events with social media.

Partnering with a Brazilian organization allowed us to systematically evaluate its application in a real-world setting in 2023.

Our partner

Fogo Cruzado (literally “Crossfire”) monitors events of gun violence in four Brazilian cities.

A small team of analysts track social media posts 24/7.

They have been interacting with users who report gun violence on Twitter/X for years

Keyword-based search with geographical filters on Tweetdeck.

Questions

RQ1 - Can Transformer-based language models accurately identify gun violence reports in Brazilian Portuguese social media texts?

RQ2 - What are the advantages and challenges of adopting a language model for real-time monitoring compared to manually reviewing social media texts?

Methods

RQ1: BERT model trained on past analysts’ interactions with tweets reporting gun violence.

RQ2: We analyzed surveys, interviews, and interaction metrics (with a diff-in-diff model) before and after the adoption of our prototype.

Findings

Small and open source

A BERT-based model in Portuguese achieved good performance. Inference can be performed on CPUs.

Our interface

Tweets were updated every fifteen minutes.

Signal-to-noise ratio

The prototype effectively filtered out less relevant social media content.

Interview with an analyst

“[Now] I do not have to go hunting for tweets.

Sometimes, I missed them [gun violence reports] because there were too many [unrelated] messages. During the BBB [Big Brother Brasil, an annual TV show extremely popular on Twitter], it was chaotic [. . . ]. It was literally a treasure hunt”

Fewer filters, greater scope

Our prototype removes the need for restrictive geolocation filters, allowing analysts to expand their search scope.

We estimated that analysts using the model engaged in nine additional daily interactions with users reporting events.

Limitations

The interviews and surveys allowed us to identify three major shortcomings:

  • The delay between each update: promptly updating new tweets is critical.

  • Static keywords used for search: terms need to be dynamically set to monitore live conflicts.

  • Use of text-only features: profile images also help analysts to decide if they will interact with users.

Two years later, final words

Major LLMs may be powerful, but small, open-source models remain useful.

Human rights monitoring: Academics 🤝 NGOs

The closure of social media APIs poses a major obstacle for low-resourced organizations.

References

Ayman Alhelbawy, Mark Lattimer, Udo Kruschwitz, Chris Fox, and Massimo Poesio. An NLP-Powered Human Rights Monitoring Platform. Expert Systems with Applications, 153, 2020. ISSN 0957-4174. https://doi.org/10.1016/j.eswa.2020.113365.

Yash Pilankar, Rejwanul Haque, Mohammed Hasanuzzaman, Paul Stynes, and Pramod Pathak. Detecting Violation of Human Rights via Social Media. In Proceedings of the First Computing Social Responsibility Workshop within the 13th Language Resources and Evaluation Conference, pages 40–45. European Language Resources Association, 2022. https://aclanthology.org/2022.csrnlp-1.6.

Hoang Thang Ta, Abu Bakar Siddiqur Rahman, Lotfollah Najjar, and Alexander Gelbukh. GAN-BERT: Adversarial Learning for Detection of Aggressive and Violent Incidents from Social Media. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022), CEUR Workshop Proceedings, 2022. https://ceur-ws.org/Vol-3202/davincis-paper7.pdf.

Thank you!

📧 Email: adriano@belisario.website

🌐 Website: belisario.website

👨🏻‍🏫 Presentation: belisario.website/crossfire_paper/